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ABSTBACT r ' ■ 

The Shawano County Evaluation Project examines the 
results of programing efforts by Extension personnel, county and 
statewide, over a 15-year period. It attempts to deal with results of 
all '13 Extension program areas, which range from 4-H to programs for 
doctors, lawyers, and other professionals. The paper describes the , 
four major variables being examined, through interviews with both a 
random sample of 1,200 households and selected people known to have 
high contact, wi.th Extension. They are: (1) nature and amount of 
contact, (2) perception of helpfulness of that contact, (3) examples 
of specific results, (4) overall benefit 'framework. Th^ variables are 
designed to fit and relate information about specific programs at a 
macro level. The paper describes how a telephone interview schedule 
was developed for use both with people with^ little or no contact with 
Extension and those who had had extensive cb^ntact. The paper also 
discusses certain philosophical assumptions inherent in this approach 
to macro-- level program evaluation: (1) in^ adult education, few 
proq ji^ffl participants hold the same specific objectives, but most have 
common broad objectives, (2) perceptioiji of help and value is as 
important as, or more important than, proof x>t results being 
attained. (Author/MS) / 
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" EXAMINING RESULTS OF SEVERAL PEOPLE'S EFFORTS IN SEVERAL PROGRAMS OVER ' . 
UJ:-. • SEVERAL YEARS - ONE TYPE OF "MACRO PROGRAM EVALUATION" 

There's no new or sophisticated methodology about the Shawano Project. If it is different 
from hundreds of other projects, "it's because we are trying to manage a study which ordinarily 
would be broken up into about 30 separate pieces. 

Because of the problems of bringing 150 copies of a lon<g paper, I will not take space to cover 
any of -.the following completely, but will skim through: 

J. Introduction to the project* 

2. Macro-evaluation dimensions of th^ product 

3, Evaluative aspects of the project • . 
, 4. Approach to results data p . 

5. Development of the cross-sectional questionnaire 

INTRODUCTION TO THE SHAWANO PROJECT 

Dr. Laverne Forest, Mary Marshall (Project Assistant), and I are responding to, a request from 
an Extension District Director to examine the impact of Extension in a county. The project 
has special ES-USDA funding as a means of testing evaluation approaches. Some of the import- 
ant background thing,s about the project are: / 

1. We are committed both to studying the impact (securing and communicating data) and 
to examining how Extension staff and . Shawano leaders interpret and use the data. 

2. We are committed to getting data on all of Wisconsin ' s \3 program areas: ag, home ec, 
4-H, government development, business, engineering, health sciences, social work • 
services, communications, labor education, environmental education, liberal studies, 
and education (not correct titles) . 

3. Extension reaches people through county activities, statewide activities, extensive 
use of media, courses, meetings., workshops, committee work. Independent Study, allied 
clubs — almost every possible means. We are dealing with all of them. 

4. We are using the Bennett model aS' a general guide and as a means of examining impact. 
* We are looking at: people reached, means by which they are reached, their reactions , 

the results they feel they -have achieved, and the type and amount of benefit from 
those results. 

5. \se are looking at cumulative impact over several years , ^ . . 

6. The total project has several stages. We have completed the enumeration of Extension 
input in the county during the past 20 years. And now we are midway into two big 

^ . surveys. .(Coding is just about completed.) The two major studies (interviews witK 

^ 300 leaders and interviews with 1,200 cross-sectional respondents) will provide the bull 

' of the information. However, if necessary., we may also do some sraaller program- 
-specific studies. ' ' _ \ 

7. We have worked with about 30 Extension agents and statewide faculty 'up to this point- 
^ * 'and expect to be working very closely with them in establishing standards of per- 
formance against which to check the data and in attempting to interpret and form con- 

) elusions from .the data. 
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MACRO-EVALUATION DIMENSION OF THE PROJECT 



Each of the three of us - Laverne, >LTry, and I - are "doing our own thing" with the project, as 
well as meeting our project commitments as n te-im. My "thing" for the -past few years has been. 
pi\?j7\w: evaluation. However, because of the great deal of confusion in how people interpret 
'•'program," 1 *ve come to use the term macro-evaluation. 

It is much enster to say what m'ncro-evaluat Lon * I sn ' t tluin to be precise about what It Is. To 
me, m.un'o-eva luat ion i.\x\jliulcti focusing solely on a single project, course, or activity by an 
individual at one location. For the most part, it excludes^ the type of thing that most of us 
learned was educational evaluation when we .'did our course-work in the 1950 's or I960's. This 
'/7?t J2'(9-t:ype evaluation - individually done, proj ect-7f ocused - is important and is very useful 
to the individual who conducts the project or course, but it serves a different purpose from 
macro-evaluation, 

Macvo-evaliiation -is^ the evaluating of the whole formed by several pieces. It does one or 

moi'e of the following: - • , - 

1. Helps monitor progress on total mission and long-range objectives. 

2. Helps summarize the results of the work of several "people, 

3. Puts things together over space (similar programs in several locations) or time 
(several activities by the same person over a period of years). 

This type of evaluation is particularly needed by state and national-level Extension administra- 
tors, although it is also essential to . the individual Extension person who deals with'- 3- to 10- 
year blocks of time and. does more than live for the moment. The top administrator's need for 
results information * to support and defend budget has beep the main impetus, both for evaluation 
.and in training faqulty in evaluation. This need has bepn poorly met by past evaluation teaching 
The diagram below shows several possible loci for evaluation. Most of us think only of tlie area 
of the chart marked A - an individual evaluating a particular course, project, or activity, in 
one location. 
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^ Community refers both to specific geographic locations and to "community' 
in the sense of like people geographical ly dispersed, ^ 



There is some disharmony here. True, results for all of the other" loci come nbouL through 
combinations of A's. However, the most efficient way of evaluating or of reporting; does 
not come from piec,ing together reports done independently by a large number of A*s. Not 
only is such an 'Effort terribly time-consupiing and frustrating, but often the report misses 
a quality which is the sum of the efforts. 

In the skawano Project, we are- -focusing on the county and community level and seeing 

how the individual pieces fit together in the following wholes: 

1. The work of one agent over a period of time (individual - county). 

2. Agents* and specialists' acti vi ties ( th and out of county) in a particular 
program area (team and program area - county) . 

3. The combi ned efforts of all program areas (all of Extension - county). 

Extension's impact on various ''communi t ics" within the county, both geographic 
and demographic (all of Extension -community). 

In addition, we will be trying to put together the whole of Extension contact of individuals 
and households over a period of time, ^ ^ 

We are trying for simple methods, and' hope that some will prove sufficiently practical and val-; 
uable that they can be adapted to other situations. 

. . EVALUATIVE ASPECTS OF THE PRO0.ECT ; 

Much of the project will be descriptive, describing amo\int of ccintact, proportion of various 
clientele reached, nature of results perceived, etc. 

' To. be evaluative, a project has to involve the making of judgments. "^Vlhe Shawano Project is 
exploring three dimensions of evaluation. 

1. We are asking respondents to make Judgments about the helpfulness of their 
cpntacts with Extension and about how Extension operates. 

2. We are asking Extension personnel and lay leaders to set standards of performance 
(what percentage should) against which to compare data. 

3. We will be helping Extension personnel and Shawano leaders forrp' judgments based on 
the data. v 

' APPROACH TO RESULTS DATA . 

Most people associate evaluation v/ith evaluation of results. Some define i;esults specifically 
in terms of changes produced by education. In terms of looking at changes produced through 
Extensiion contact, we are Rising the following: 

■ 1. open-ended responses to questions which ask fqr examples of how Extension informa- 
tion has been used. We will ^look for patterns. 

2. ■ judgments of amount of benefit gained in relation to a general benefi t framework: 
economic, envi ronmental , physiological , socio-psycholog ical , socio-pol i tical , and . 
educational resources. 

The first approach will help us determine whether, from the clientele's point of view, im- 
portant results do coalesce and mass around certain objectives held by Extension personnel, 
or if result is unique to each participant. 

The second will help us summarize a variety of very specific kinds of results into major result 
For example^ although the nature of the ability is different, the lawyer, the A-H leader, and 
the farmer^ dre all helped to develop their own talents and abilities (socio-psychological 
O fit area) . 
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DEVELOPING THE CROSS-SECTIONAL STUDY 

The major data input comes through two surveys. I've worked with the cross-sectional study 
and will share a few. of our experiences . 
*• 

1. We met .with "think groups" representing Extension program areas and explored what informa- 
tion they wanted. We tried to get them to identify decisions they needed to make, but 
were not very successful. 

2. Our first ideas involved about 30 separate sub-studies. On the advice of our national ad- 
visory group, we decided that we could probably get the crucial information through a cross- 
sectional study. ■ ' 

3. We decided on interviews with a minimum of 1,000 households. The main factor in choosing 
this large a sample (the population of the county is only 35,000) is that we wanted a sample 
of about 100 farmers, and farmers make up about 1/5 of the population. We: 're not sure if we 
wi have eiiough elderly or Native Americans out of the 1,000 or whether we will have to do 
some further interviewing specifically with these, or with other potential clientele who 
appear in small numbers within the county's population. 

4. The actual telephone survey was contracted to Survey Research Laboratory. We worked with 
their staff in design, instrument development, etc. 

5. Their experience showed that it is possible to complete accurate telephone* interviews which 
go as long as 45 minutes or an hour if the respondent is comfortable with the questions. 
Telephone survey proved less co5^tly and fully as productive as in-house surveying. 

6. After putting everything everyone wanted to know together, we found the instrument went 

way beyond an hour. We .split it into two versions, using^ome of the same questions in each, 
but different detailed questions. Of course, we weede^out a lot of low-priority questions. 
"Skip-outs" became a crucial phenomenon, moving a person^who had not had a particular kind 
of experience on to another lead question: for example, skipping the non-4-H person over a 
page of questions related to 4-H . 

7. After the pre-test, we found the instrument was too' much of a burden for'people who have had • 
little or no contact with Extension. ' So we developed a third, much-reduced schedule - 
Schedule C. ' / 

The'cove^ sheet (introduction to the . interview) asks the respondent how much contact- 
. (s)he has hnd with Extension. Schedule' C is used with those who respond "Little", or "None." 
The cover sheet directs the. interviewer to .either Schedule A or Schedule B if the response 
is "Some" or "Much." 

About 20 different people did the interviewing. They tended to stay with either the A 
or B schedules. -V 

Schedule C,- however, asks two pages of questions about Extension contact because we are aware 
that people often do not remember or recognize that a. contact is Extension. We ask about- 
4-H or about contacts with agents both by name and by '^position.. We expect we-will find a 

percentage who said they had uo contact, but on later 'exploration actually have had contact. 

\ . ' ' 

\ 

\ 

The survey was completed in mid-March with a total of 1 , 192 interviews or an 80. 5% response 
rate. Of the interviews, 174 were on Schedules A or B.. (These schedules were about 34 pages 
long -and took .about an hour.) The C Schedule was 13 pages lo'ng and took about 20 minutes. 
(2,545 telephone numbers were called, but 1,065 were non-sample cottages, businesses, or not 
in Shawano County.) 

I'm 'sorry that it is too early to talk about results of the approaches used. However, if some- 
thing mentioned here -interests you, get in touch. 1 .. 
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